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A study assessed the applicability of the American 
Institutes for Research (AIR) procedures' 

Performance-Against-Standard form for use in. personnel evaluation by 
the Ohio Cooperative Extension. Service . The study also compared 
relevant test construction approaches identified in the literature to 
those used in the AIR study in attempting -to confirm .and improve the 
reliability and validity of the performance appraisal form. During 
the study, supervisors of Ohio's 10 extens.ion areas each appraised 
the performance of those agents working under them who had been 
employed for at least 6 months but for lers than 3 years. Job 
analysis and the technique of item selection were used to judge item 
availability on the form. To measure the reliability of the form, 
researchers utilized Cronbach' s- alpha.. In addition, multivariate 
measureme.nt was used to determine the accuracy and relavance of the 
form. Because evidence indicated that the revised 
Per-f ormance-Against-Standard form does provide an accurate and 
relevant measure of employee performance, recommendations were made 
calling tor including the form into the appraisal system of the Ohio 
Cooperative- Extension Service. (The six page personnel appraisal form 
is appended. ) (MN) 
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The accurate appraisal of a county extension agent (CEA) is a 
major concern of the Cooperative Extension Service. Normally., per- 
formance appraisal is undertaken for two purposes — judgmental and 
developmental (Cummings and Schwab, 1973). 
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Judgmental or evaluative appraisal provides results f6r adminis- 
trative decisions concerning such aspects as rewards Dased on merit, 
promotions, transfer and termination. Developmental appraisal provides 
results to help improve performance by identifying areas for improve- 
ment and growth. Thus, an effective personnel appraisal system focuses 
on both past performance and on improvements in future performance. 

Historically, the Cooperative Extension Service has utilized a 
variety of techniques to appraise the job performance of the county 



extension agent. These methods generally have been subjective in nature 
and have ranged from personal interviews to subjective ratings. -In 
addition, measurement of individual abilities to perform on the job ^ften 
have been based on various tra,its or characteristics of employees which 
have little relationship to successful task performance (Cununings .and 
Schwab, 1973 )• Consequently, the effectiveness of such methods has been 
restrained by a lack pf evidence on the accuracy of the judgments of the 
appraisers. 

In recent years, testing procedures have been developed and utilized 
to bring more accurate measurement into the extension performance appraisal 
process. A major advantage of testing is that it provides one with a 
standardized and objective judgment of what an existing employee has 
accomplished. In addition, the scientific evidence of reliability and 
validity pr^ocedures derived from using standardized measures provide more 
useful information than do traditional subjective methods in confirming 
the actual effectiveness of the performance appraisal procedures used by 
the organization. 

One testing procedure used to measure employee .performance that is 
gaining support is to base performance measures on job analysis and general 
knowledge, skill, and competency related to human behavior (Hyatt, 1966; 
McCormick, 1979). The American Institutes for Research (AIR) has contrib- 
uted substantially toward this effort by utilizing the' component of work 
behavior in the design and construction of standardized tests to be used 
in performance appraisal at .different levels and positions of the 
Cooperative Extension Service. 

In 1977, the Science and Education Administration (SEA -Extension) of 
the U. S. Department of Agriculture employed the American Institutes for 
Research (AIR 5 to develop standardized tests that cpuld be used by state 
Cooperative Extension Services in the selection of new employees as county 
extension agents and for performance appraisal of existing personnel 
employed as county extension agents. To accomplish this goal, AIR con- 
ducted a nationwide study entitled ^"Development' of Performance Evaluation 
and Selection Procedures for the Cooperative Extension Service" (Hahn, 
1979).. As a result of the AIR study, a personnel appraisal form entitled 
"Performance-Against-Standards Form" was developed and provided to the 
Cooper?ative Extension Services 

Because different job duties and specifications exist among various 
state Cooperative Extension Services, AIR recommended that each state make 
its own decision about using the pierformance appraisal form in accordance 
with the state's objectives and focus of management philosophy. While it 
is necessary for each state to consider the applicability of the form, it 
is necessary also to consider the accuracy and relevance of the form. 

It is the purpose of this study to determine whether the personnel 
appraisal form designed by AIR has acceptable levels of reliability and 
validity for use by the Ohio Cooperative Extension Service. Because the 
reliability and validity levels of any performance appraisal form v^ill 
vary in response to the procedures used to construct the form, this study 
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also will compare the procedures used by AIR in constructing the performance 
appraisal form to existing research and theoretical considerations. 

The validation procedures to be used in this study are as follows. 
First, job analysis, item selection based on item-total score correlation, 
and a scoring system measured by a linear summative scale will be applied 
ta confirm the effective test construction of the personnel appraisal form 
developed for Cooperative Extension Service by American Institutes for 
Research (AIR). Second, the reliability of the form will be estimated by 
the* method of internal consistency. "The alpha coefficient will be the 
major criterion used to indicate the degree of interr^latedness among 
items on the form. Finally, the construct validity of the performance 
appraisal form will be determined through correlation and factor analysis. 

AMERICAN INSTITUTES FOR RESEARCH PROCED.URES 

Job Analysis 

Job analysis attempts to describe a job in terms of the demands made 
upon 'the job incumbent (Cummings and Schwab, 1973). A rigorous procedure 
of collecting information and materials in support of a jdb analysis was 
carried out by AIR. The job analysis consisted of five methods: inter- 
views, critical incidents, position analysis questionnaire, job analysis 
questionnaire, and checklist of behavioral and other work outcomes. The 
classification of job tasks, job duties, and performance dimensions were 
determined by subjective judgments of supervisors and agents who partici- 
pated in the study. Also, the critical incident technique for collecting 
joL outcomes was used as, a base for a discussion of knowledge, skills, 
attitudes, and other characteristics (KSAO requirement) necessary to the 
^ successful job performance by»extensioji agents. The job analysis provided 
two import ant, out puts of KSAO contents' and constructs to be*^ tested and 
performance domains to be measured on the job. The two output contents 
are considered to be basic requirements for the county agent position and 
provide fundamental item clusters used in formulation of the Performance- 
Against-Standard Form. A brief review of the AIR test construction 
procedures' is '.provided in the following sections. 



Item Selection » - 

The elements of job outcomes lised by AIR to construct the Performance- 
Against-Standard Form were collected in the stage of job analysis. Fourteen 
dimensions of performance outcomes originally were classified by subjective 
judgment of supervisors and agents. These dimensions are presented in 
Tabie 1. ' , ^ 

There were 1,003 items or descriptors structured around fourteen 
dimensions and included^ in preliminary checklists describing behaviors and 
results. AIR respondents were requested to scale each descriptor on a 
seven-point scale. » The average rating assigned the descriptor identified 
the degree to which the descriptor or standard described effective 
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Table 1. Dimensions of Performance Outcomes 

^as Classified by Supervisors and Agents 



1 . 


Assessing community needs ' 


2. 


Planning 


3. 


Program promotion and public relations 


n. 


Involving the community in program implementation 


5. 


Conducting education firograms 


6. 


Advising clienteles ■ * 


7. 


Evaluating programs 


" 8. 


ReDortinst 


9. 


Continuing professional development 


10. 


Office management and administrative tasks 


ii. 


Supervising ^ 


12. 


Working relations with staff . 


13. 


General interpersonal behavior land characteristics 


14. 


Personal. J>ehaviars and characteristics as identified in 



AIR study 



performance on a dimension. The standard deviation of the rating for each 
performance standard (descriptor) represented the amount of. agreement among 
the raters. Each preliminary item was tested against the following , 
criteria. « , , ^ 

1. An outcome had to have been observed by 80 percent or more of the 
supervisors, . . 

2. * An outcome had co be applied to each job class in the opinion of 

80 percent or more of the respondents. 

3. The standard deviation of the scale value for an item could not 
exceed 1.50. 
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The item distribution of scale values could be multimodal. 

5. There could be no statistically significant difference between the 
supervisors and agents j.n their scale value for the item. 

^ • - ^* 

There were 7^9 items that met theSe criteria and were retained as an 

item bank of performance outcomes. Of this number, 200 items were selected' 

■fo meastire'*'the six categories of the Performance-Against-Standard Form and 

summarized in Table 2 (Hahn, 1979:25). 



Table 2. - Functional Categories and Number of. Items in the 
Performa'nce-Against-Standard ..Form 



Functional categories 


Result-oriented 
standard* 


•Behavior -oriented 
stanaard 


Program planning 


10' 


15 


Program promotion 
.and public relations 


10 


i . - 

10 


Program ^implementation 




.. 35 


Program 'support 


10 


25 


Interpersonal and personal 




"20 


Supervisory perforinance 




-20 ■ ' 



It should be noted that each functional category listed in Table 2 
may cover several performance dimensions and relate to several job duties. 
A comparison of Tables 1 and 2 illustrates the overlapping of function and 
dimensions. The function of program planning includes dimensions 1 and 2; 
the function of program promotion and public relations covers dimension 3; 
the function of program implementation consisting of dimensions 5, and 
6 J the function of program support covers dimensions 7, 8, 9, and 10; the 
function of interpersonal and personal included dimensions 12, 13, and lU; 
and the function of supervisory performance is associated with dimension 
11. In sum, the original .Performance-Against-Standard Form is more cor- 
rectly defined as including 10 componeftts which were generated from 6 
functional areas. As*lpi?esented in Table 2, U components are result- 
oriented standards and 6 components are behavior-oriented standards. 
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Scoring 



For each of the 200 items invthe Performance-Against-Standard Form, 
5 categories (agree, disagree, neither, don^t know, and not applicable) 
were provided to enable the rater to appraise the ejnployee^ In addition, 
^% predetermined weight was assigned to each ite.m based on the judgmen;t: of 
a panel of 80-100 agents and supervisors (See Appendix A). A description 
of the AIR scaring procedure is provided. .... * 

For all standards having a positive weight, record th6 value 
of the standard which appears to the right of the statement 
in the scoring space provided, if there is a checkmark in the 
agree column, . or an or B>> in the disagree column* If there 
is an L'^ or W»^ in the disagree column, record a zero in the 
scoring space for the standard. - • 

For all standards having a negative weight, record the value ^ 
of the standard, disregarding the minus sign,* in the scoring 
space provided if there is an H or a B in the disagree 
column. If there is- an L or W in the disagree column, record 
a zero in the score space. Also, if there is a checkmark in 
the agree column, record a zerp in the scoring space.- ' ^ 

* The scorning space is to be left blank for any standards that 
have responded in the neither, don^t know, or not applicable 
columns. Computje the Performance-Against-Standard scores 
\for each seation by adfling the scores, divided by the sum of 
the values of items scbrjed in the section, disregarding sign^' 
carrying out the quotient to two decimal places, and multiply- 
ing by 100. l*ho division and multiplication by 100 is 
necessary where the standards are used outside this example 
because some agents may be rated on fewer items than others. 

*'«H = Higher; B = Better; L = Lower; W = Worse 

^ Reliability , , 

^ reliable determination of each item /.n the AIR study was based on . 
the criterion of not exceeding a standard deviation value of 1.50./ This 
reliability reflects the relative amount of agreement among jildges as to 
the scale location* of a particular item. 

<. 

A second measure of reliability was Cronbach^s alpha. However, the ' 
study reported the reliability of the only one dimension (conducting educa* 
tional programs). Its value was .99. No further evidence was provided to 
estimate reliability.. ^ -i 

Validity 

, AIR reported that the Performance-Against-Standard Form contained a 
high intercorrelation for the fourteen dimensions and the overall rating 
of job effectiveness. . However, evidence of construct validity was not 
provided by AIR. - • 
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TEST CONSTRUCTION PROCEDURES 

The major purpose of this study is to estimate the applicability of 
the Performance-Against-StandardTorm for use in personnel appraisal .by 
the Ohio Cooperative Extension Service. This study also compares relevant 
test construction approaches identified in the literature to those used in 
AIR's study in attempting to confirm and/or improve the reliability and 
validity of the performance appraisal form. The^ selection of other test 
construction approaches is based on existing research and theoretical 
considerations (Shiao, 1981). 



Item Selection 



In the AIR study, the selection of items to be included in the' 
Performance- Against-Standard Form was based on two major criteria. The 
first criterion used to discard items was that an item had to be confirmed 
.by 80 percent or more of supervisors as being applicable to performance 
appraisal. ^Second, the standard deviation of the scale value for, an item 
could not exceed 1.50. 

The criterion of applicability of 80 percent is consistent with the 
literature on development of a performance appraisal form and also is 
adopted for use in the present study. Some researchers, however, are 
opposed to using the standard deviation to retain items (Smith and Kendall/ 
1963; Campbell et al. , 1973; Fogli, Hulin, and Blood, 1971) because the 
standard deviation emphasizes the agreement within a group of judges but 
does not consider the relationship of each item with its structural 
dimension for confirming the expanding item ''validity. To increase the 
homogeneous structure of the Ferformance-Again^t-Standard Form and to re- , 
evaluate the item validity of each item, the criterion of item-total score 
correlation will be used in this study to confirm 'the selection of items 
included in the Performance-Against-Scandard Form. 



Scoring 



A predetermined ccore was established Tor earch item and. each dimension 
in the Performance-Against-Standard Form by AIR. Empirically, the use of 
a pre-weighted scorje could influence the unstandardi^ed judgment of the 
criterion group. The ambigubus definition of successful or unsuccessful 
perfqrmance could. increase judgmental bias in selecting items. In addition^ 
*the different weights assigned to behavior patterns could be judged from a 
variety of perspectives (Schwab et al., 1975; Cummings and Schwab, 1973). 

^ A second limitation of the pre-weighted scoring system is that agents 
may obtain zero value for an item if their performance is considered below 
the state level of that performance standard. It may be argued that one^ 
cannot perform at zero value in a behavior-oriented job outcome. An 
alternative scpring measure will be used in this study and is similar to 



ERLC . . . 



. . 8 

the scoring measure of a summated scale. Use of a summated score based on 
an equal-interval scale should diminish the bias of a reference group 
because each item will be assigned equally weighted value alternatives. 

A third limitation of the scoring system use?i by AIR is that the ^ 
weighted value assigned to each dimension in the AIR study was derived 
from the classification of subjective judgments and based on the relation- 
ship of categories of job duties. This procedure of weighting dimensions 
does not consider the intercorrelation among the components. .^In fact, no 
evidence was provided for the weighted value assigned to each. component in 
the performance appraisal form. The confirmation of weighted values for 
each of the ten components will be conducted ,via factor analysis. Strictly 
speaking, the emphasis on assigning a weighted value to each component is 
considered to be more rational and have more^ practical meaning than to 
assign weights to each performance item. , 

In sum, the alternative scoring system used in this# study is based on 
the following, considerations: . ^ ^ 

1. The irrational problem of providing a zero score to measure performance 
outcomes would be diminished. 

2. The measurement of behaviorally oriented items would be considered to 
. provide the continuum of a scale value. 

3. The alternative system would permit the addition of new items to 
the Performance-Against-Standard Form ba^ed on the need of varied 
job situations. * 

The summated scores of each component and the overall performance 
score .will be used to represent the obtained scoi?es of each agent's^per- 
formance. Since each agent *s score may be based 'on responses to different 
items in terms of applicability, an average summated score will be pre- 
pared. That is, the total score will be divided by the- number of items 
scored. 

• . " i ' ■ 

Reliability . 

AIR provided evidence to support the reliability of only one of the 
dimensions jof the Performance-Against-Standard Form. The reliability 
of item structure of each dimension of the form will be examined in thip 
study. ^ ' * , ^ 

Validity ' ' . 

Only limited evidence of validity was reported in the AIR study. The 
classification of each job dimension was subjectively evaluated by profes- 
sionals and no empirical construct validity explanation was provided by 
the AIR research. Ensuring the validity of the construct structure of job 
performance outcomes and components in the Performance-Against-Stahdard 
Form will be undertaken, in this study. 



METHODOLOBY 

The data- for this study were obtained from the supervisors of the ten 
extension areas of Ohio, Each extension sufJervigor appraised agent per- 
formance in the early spring of 1980. The analysis of data was conducted 
at the Institute and Research Computer Center ax The Ohio State University, 
SPSS and SAS computer packages were used in completing the statistical 
analysis, , ' ^ 



Sample 

The sample of this study consists of. county extension agenta in the 
State of Ohio whq met the following criteria:^ 

1. The agent must have worked at least six months but not more than, 
three years. The employed time would be from spring 1977 to autumn 
1979. I 

2. The agent must still be working on the job_ during the period of the 
study; 

M easurement and Scoring 

As reported in Table 2, the AIR Performance-Against-Standard Form 
consisted of 200 items classified into ten component sections. Since the 
agents included in this 'study had no supervisory responsibility,, the super- 
visory component (20 items) was- omitted from the analysis. "Thus, the 
performance appraisal form to be tested contained 180 items. 

The scoring of each item on the form was conducted using two proce- 
dures. First, each item was scored according ,to the procedure used in the 
AIR study. Second, each item was scored using the equal-interval scoring 
system.^ The AIR scoring procedures were described previously. In sum, 
each agent received a score for some or all items within each component. 
In addition, a score for each component and a total performance score w^s 
calculated for each agent.. * 



These two criteria were developed for a larger study concerned with 
employee selection and performance appraisal. The objective of the larger 
study was to compare scores on an employee selection test (designed by^ 
AIR) to scores on the performance appraisal f orn{j. at one "point in time. 
However, an examination of selection test results indicated that the 
selection test did not provide reliable scores. Therefore, a comparison 
of Selection test results to performance appraisal scores was eliminated 
from the overall study. 

' ' ■ 10 ' > - 
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A description of the equal-ihterval scoring, procedure is as foilows: 

The value of 3 is assigned to a checkmai^k with H (KitKer) or 
B (better) in the disagr'ee column; the value* of 2 for agree 
column; and the value of 1 for disagree^^with L (lower) or " 
(worse). i , ^ 
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Item Selection - - . 

^ In utilizitig a standard ^orm for performance appraisal, it is neces- 
sary to eliminate items that do not apply to a particular individual's job 
performance. For purposes of this study, those items in the Performance- 
Against-Standard Form that were "not applicable" to 20 percent or more' of 
the respondents were eliminated from further /consideration. 

Following this criterion, the item numbers listed in Table 3 were 
eliminated from further consideration in determining the relevance and 
accuracy of the Performance-Against-Standards Form, Seven items were dis- 
carded — one from section IB, five from section JC, and one-^ f rop section IIC* 

Table 3. Items discarded* with 20 percent or more'' ' - 
- * - "not applicable" responses 



Reference No. of Itema 



% of Unapplicable 'Re^sponses " 



IB 


22 




31 


,ic 






29 


IC 


73 




'24 


IC 


H^ 


* 




IC 


65' 


« 
* 

« 


24 




106 




21 


lie 


112 




26 



n = 62 / 

^The description of each item is? contaihetl in Appendix A. 
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Table 4* Items discarded with the criterion pf coefficient of 
i,tem-component correlation 



Categories Reference Correlation Coefficient 
of Components No* of Items of Item-Component* 



-Component IIB IIB 35 
, ^ .IIB "36 



Component IIC^ 
* • " 



lie 
lie 



il8 
^9 



lie 101 
lie 103 



- j 

•Not_ signif.icant at .05 level 



Component 


IC . 


le 


- 38 


0.14 




2.02 


0.02 






le 


95 


0.26 




1 .80 


0.26 






le 


11i| 


0.31 




1.97 


0.10 






le 


' 93 


0.21 




" . 2.83 


0i32 






le 


.96 


-0.10 




. 2.90 


0.20 






IC 


97 


0.30 




1.83 


0.28 






IC 


12^1 


0.16 




- 2.92 


0.16 


Component 


ID 


■■ ID 


136 


0.22 




1.9^ 


0.06 


Component 


< 

IIA 


IIA 


21 


0.24 




2. Oil 

Ji. 


0.11 

• 



0-29 



0-23 
0.24 
-0.03 
0.16 



2-00 0.07 
2.00 O.Oil 



2.96 
2. Oil 
2.92 
2.90 



0.(?7 
0.11 
0.17 
0.19 



A second and ^ more important criterion used to select iteAis in a per- 
formance appraisal test is the relationship of indivijdual item scores to 
the total score of a component or dimensior^. The criterion measures the 
homogeneous^ support of items to the/ overall item structure by comparing 
the \3caling characteristics of the individual item to the total component 
measure. A vafid itfem should have a significant an^ positive corrf^lation 
to the total sco^ip of the overall item structure. A positive correlation 
with a' statistical significance* at^ the ,0.05 .level was used as a standard 
fp2? judgment of Homogeneity of ftems^ Table ^ lists thos^ items which did 
not reach the acceptable standard.' 

In sum, 'a .ih.avised item structure, of each component — after taking out 
ineffective items — has been cbnstijucted. The number of items in the revised 
Perforfrtan^^-Against-St^ndard Form.ar^ compared to the original number of 
items in\each dimension iij* Table 5. The elimination of ineffective items 
reduces ,the form ^rom 180 to 1*12 i1:ems. ^ * - 
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Table 


5« Comparison of item number between revised and original 
dtem structure of Performance-Against-Standard Form 


• 


Component Categories 


• • ^ 
Number of Items 
on Revised Form 


Number of Items 
on Original Form 


(lA) 


Program planning "Results 


10 




10 


(13) 


Program promotion and 
public relation results 


9 




. ro 


(IC) 


Program implementation 
results 


33 






(ID) 


Program support results 


< 




10 


(IIA) 


Program planning behaviors 






15 


(IIB) 


Program promotion and 
public relation behaviors 


8 




10 


-<IIC) 


Program implementation 
behaviors ^ 


27 




35 


(IID) 


Program jiupport behaviors 






25 


(XIE) 


Interpersonal and personal 
behaviors generally , 
related to Job 


15" 

r 




20 




Total' 


142 




180 



• Roliability 

To romovo inoffoctivo iton)S is the firat ntop to incroafling the roln- 
vancy and accuracy of iho PorformancorAgalnat-St^ndard Torm. . A oocond fUcnp 
is confirmation of tho dogroo of roliability of the form. Tho calculation 
of Cronbach's alpha as a criterion of roliability providon support for the 
dogroo of accuracy of tho porformanoo appraioal form. Cronbach'n alpha 
moaauros tho avorago coefficient of In tor-item correlation, and reproyent^ 
tho degree of internal conoiatency of tho performance appraisal form. 

Table 6 compares tho roliability of tho Porformanco-Against-Standard 
Form before and after discarding ineffective items within each coi^ponont 
baaed on the- use of tho omplrlcal-lntorval, scoring.^proceduro. The table 
. roveals that the i?oXlaBility_of form ia not oub- 

Stan tidily changed wFth the removal of ineffective items. Thus, one may 
as'^umo that the procedures used for item retention are viable. / 
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Table 6.. Compar*ison of Cronbaoh-alpha reliability of the 
J Performanoe-Against-Standard Form before and 

3 after discarding ineffective ' items (using 

empirical-interval scoring procedure) 



Reliability Reliability 

Before ^ After 

Discarding Number Discarding - Number - 

Components* Items of Items Items of Items 



(lA) 


Program planning 
results • 


0.78 


10 


0.78 


10 


(IB) 


"Program promotion 
and public relation 
results 


0.58 


' 10 


0.55 


9 


(IC) 


Program implementa- 
tion results 


0.93 




0.92 


3*1 


(ID) 


Program support 
results . ' 


0.71 


10 


0.7^^ 


9 


(IIA) 


Program planning 
behaviors 


0.90 


■ 15 


0.90- _ 


14 


(IIB) 


Program promotion 
and public relation 
behaviors 


0.69, 


10 


0.69 


8 


(IIC) 


Program implementa- 
tion behaviors 


0.93 


.35 


0.92 


28 


(IID) 


Program Xupport 
behaviors 


0.86 


25 


0.86 


17 


(HE) 


Interpersonal and y 
personal behaviors 
generally related 
to job 


0.88 


20 


0.87 


15 



•H 



Tho roliability of oach componont In Table 6 also showG that tho 
Internal consiutoncy of compononts is difforont. Compononts IC and IIC 
aro concornod primarily with program implomontation roaulto an^l behavior 
and have higher roliabiUty coefficients than do tho other components* 
The rqliability of component IIA is higher than tho component of program 
planning results (lA) and has the second highest loyel of reliable oyidonce. 
As a general- rule, a roliability coefficient of .80 or bettor is desired. 
However, Components lA, IB, ID, and llB have lower reliability coofficlents. 



Scoring System 

The degree of roliability of tho Performanco-Against-Standard Form 
also is affected by tho scoring system used.' The property of a scoring 
system is composed of arrangements of scale points, weighting score for 
qach item or o^ch dimension, and calculation of a total score. Since each 
of these properties can have an affect on reliability of , the performance 
appraisal form, it is difficult to determine in advance which scoring 
system should bo used. It was reported that the AIR study assigned a 
pre-weightod score to oach item in the performance appraisal form. This 
study compares the reliability of that scuiing system to one that utilisses 
an equally-weighted- value scoring system. 

Tho data shown in Table 7 gives the comparison level of reliability 
obtained when using empirical-interval score and the pre-weightod score. 
The alpha coefficient in the empirical- interval score is higher than in 
the pre-woighted score. Since both scoring systems have the same item 
structure and number of items, the use of the empirical-interval scoring 
system appears to be more internally consistent thaii does the pre-weightod 
scoring system* 

One explanation of lower i'>oliability for the pre-welghted score used 
by AIR is that tho pre-weightod score does not discriminate between average 
and superior performance. That is, agfes who meet tho minimum standards 
are given tlie same score as those agents who exceed minimum performance. 
Thus, while the pro-weighted scoring system may bo tho dominant design to 
datermino a weighted value for each item, in representing successful or 
unsuccessful job performance, its use is limited to a unique job pattern 
and job time and is difficult to use under different job situations and 
changing programs. 

/ ^ 
^ In conclusion, the evidence reveals that tho empirical-interval score 

provides a higher reliability coefficient than does the pro-weighted scor- 
ing system. Because the empirical-interval score provides the advantage 
of allowing additional items to increase the reliability of the performance 
appraisal form without changing tho scoring system, use of empirical- 
interval scores to measure job porformancas is recommended. 
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Table 7- The level of reliability "of empirical^interval scoring 
system and pre-weighted scoring system 

Empirical-Interval 
^ Score 

, Three-Scale 

^ \ ^ ^ Point Pre-Weighted Number 

Component Categories (3,2,1) Score of Items 



(lA) Program planning 0.78 0.65 

results 



tHD) Program support 0*86 
behaviors 



(HE) Interpersonal and - 0^87 0.83 
personal behaviors 
generally related 
to job 
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(IB) Program promotion 0.55 0.5^ q 
and public relation 

results ' 

(IC) Program implementa- 0.92 0.89 
tion results 

(ID) Program support 0*7^1 o.6l 
results 

(IIA) Program planning o»90 0.76 
behaviors 

(IIB) Program promotion 0.69 0.^9 
and public relation 
behavior- 

(IIC) Program implementa- O.92 0.84 
tion behaviors 



3M 
? 

8 



28 

0.70 J7 
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Tables. Pe£raon correlation uong iteat >y perfomance appraiaal components lA, IB» IIA 



lA*. ProgritB Flahnlng Keaulta 



IB. 'FrograK Froaotion and 
Public Relation Keaiilta 



IIA. Frograa Planning Behaviora 







12 1 


2 


19 


7 


18 


16 


4 


17 


i: 


35- 


38 27 


28 


26 


29 20 


34 


32 


24 


19 


18 


9 


6 


4 


3 


10 75 


8 5 


' 1 


11 


17 




12 


1 .32 


* 


.39 


•41 


.51 


.45 


.42 


.23 


.22 


.51 


.48 * 






* .57 


.4.9 


.48 


.57 


.57 


.35 


A 




A 


A 


* .58 




.29 




.51 




1 


1 




* 


* 


* 


A 


.40 


A 


A 


.47 


.33 * 






A A 


A 


.34 


.47 


^46 


.68 


.55 




.34 


.36 


* .35 




.44 




.41 




2 




1 






A 


A 


A 


A 


A 




A A 




.40 




* 


A 


A 


.33 


A 


A 


.32 


.34 




* .64 


*. .32 


.42 




A 




19 






1 


.47 


A* 


A 


A 


A 


.29 




A A 






* .40 


A 


A 


A 


.47 


A 


A 


.31 






* .48 




.38 




.34 


lA 


7 








1 


A 


.43 


.33 


A 


.43 




A. A 






* .38 


.30 


A 


A 


.58 


A 


A 








* .49 




.30 


.*51 


.39 
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1 


.47 


.37 


A 


A 




A A 






* .38 


.30 


.51 


.35 


.37 


A 


A 








* .36 




A 




A 
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1 


.39 


A 


A 




* .51. 






* .38 


A 


.48 


.34 


.54 


A 


A- 








A A 




A 




.43 




4 














I 


.41 


A 


.32 


.28 .49 






* .6: 


.41 


.43 


.46 


.64 


.39 


.49 








* .45 




A 




.44 
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1 


A 




.42 .38 


.35 




* .36 


.33 


.31 


.46 


A 


A 


.40 








* .33 




.30 




.28 
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1 




A A 






A A 


.35 


A 


A 


A 


A 


A 








A A 




.27 


.28 


A 



38 
27 
28 
26 
29 
20 
34 
32. 
24 
19 
18 

9 

6 

4 

3 
10 
25 

8 

5 

1 

11 
17 



.31 
1 





* ■ * :45 


.*9 


— r 


.27 


* * .51 


A 


.45 


A 


AAA 


A 


A 


1 


AAA 


A 


A 




1 .85 * 


A 


A, 




1 * 


A 


.31 




1 


.42 


.64 






J 


.41 



.41 

A 
A 
A 
A 

.45 
54 

56 
I 



.30 

A 
A 
A 

.29 
.65 
.57 
.64^ 

3r 



* .40 

A A 

A A 

* .39 

A A 

A A 

A A 

* .35 .30 



.41 M 
.32 * 
1 * 
1 



3F 



.32 

A 
A 

A. 

.36 
.49 

A 

.50 



746 



.35 



A 
A 
A 

.32 
.37 .44 
* .32 

A 
A 



.29 


.30 * * 


.53 


M 








.« 


.44 




.54 


.40 




.35 


.38 


.65 


A 




A 


.38 




.36 




.33 


A 


* .45 ' * 


. A 


.37 




A 




A 


1 




.56 


A 


.41 


.30 


.31 


.41 




J ' A A' 


A 


.43 


A 


.34 




A 




1 .31 


A 


A 


.40 


-.32 




.29 




1- 


A 


A 


.48 


' A 


.39 


A 






1 


A 


.38 


.53 


.33 


.53 








1 


A 


.35 




.45 










1 


.58 


.37 


A 












1 




.33 



740 

A 
A 
A 
A 
A 

.58 
.34 
.47 



•22 
1 



N > 62 

*Ho significance at the .05 level 
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Validity 

The Qonstruct validity of a performance appraisal form ensures the 
appropriate construct relationship among dimensions (components) and 
enables one to better interpret the overall construct structure. 

Theoretically, the number 'of items "designed to measure a construct 
should correlate highly to the item structure of each component (construct) 
and have low"^ correlation or no correlation with items which measure 
another construct and considered to be independent of the first construct. 
Consequently, the definition of a .construct as a unique variable is con- 
firmed by the degree of relationship of one construct to other constructs* 

In the AIR study, the Performance-Against-Standard Form was composed 
of nine components generated from 14 performance categories. If the 
dimensional patterns of the components in the Performance-Against-Standard 
Form are independent of each other, the correlation between components 
will pot be significant and the Performance-Against-Standard Form can be 
treated as a multi-dimensional oi? multi-criterion form. Generally, a 
multi-dimensional measurement instrument is desired. 

Items used' to measure a construct .should be related to other items in 
the same construct but should not be related to items in other constructs. 
Since the performance appraisal form contains nine components (supervision 
component was not applicable to this sample) and 1U2 items, it would prove 
tedious reading to present intercorrelation among all pairs of constructs 
and items. Therefore, the discussion of findings wilj. be restricted to 
the item correlations within and between constructs of three components 
(lA, IB, and IIA) and presented in Table 8. 

In Table 8, the items of component lA (program planning results) 
should be highly correlated to each other and should have low correlations 
with the items of component IIA (program planning behaviors). In addition, 
components lA and IIA should not be highly related to component IB if the 
relationships among the three components are independent. The examination 
of Table 8 reveals that correlation of items within a component are similar 
to correlation of items between components. 

Since the paired comparison of items between all nine components is 
quite lengthy, an average score for each component was computed. Correla- 
tions between components were „ than calculated and are presented in Table 9. 
All components ar^. significantly related to each other and to the average 
score of overall performance. In addition, t\\e size of the correlation 
coefficients indicates that the scoring tendency of each dimension tends 
to unfold with the total score of the performance appraisal form. The 
evidence further confirms that the Performance-Against-Stan^dard Form has 
com posite s tructure and nat a multi-dimensional structure. 

Due to the composite nature of the Performance-Against-Standards Form, 
common factor analysis was used to determine whether performance appraisal 
components^ pre$ented in the correlation matrix of Table 9 can be accounted 
for by one common factor. Common factor analysis assumes that the variance 
of an i*feem variable (or component) is common to other components. Unrotated 
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Table 9: Pearson product moment correlation matrix by 
*^ component and overall score for Performance- 
Against-Standard Form.* 



Component 






















T A 


TR 




TH 
XU 


T T A 


TTO 




T T n T T r 

IID HE 


X+ 


lA 


1.00 


















IB 


.65 


1.00 
















IC 


.78 


.66 


1.00 














ID 


■ .65 


.47 


.69 


1.00 












IIA 


■.^9 


.39 


.61 


.53 


1 .00 










JIB 


.59 


.5^1 


.47 


.39 


.43 


1.00 








lie . 


.69 


.59 


. .69 


.60 


.64 


.59 


1 .00 






IID 


.51 


.H6 


.60 


.58 


.60 


.34 


.58 


1.00 




HE 


:69 


.66 


.69 


.51 


.56 


.51 


.56 


.44 1.00 




X+ 


.85 


.74 


.90 


.67 


.73 


.66 


.86 


.74- .78 


1.00 























n s 62 

*A11 correlations are significant at .01- level 

X+ is average score of overall Performarice-Against-Standard Form. 
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orthogonal (principal axis method) and promax oblique rotated methods of 
factor analysis were employed (the sum of all item loadings on the ' 
factor). Kaiser (1974) suggests that eigenvalues greater than or equal 
to one should be retained. Following this rule of thumb, the factor 
matrices presented in Table 10 indicate that all nine components are 
patterned as a unique factor. ' , 

The square of a loading multiplied by 100 gives the percent variation 
that a component has in common with a common factor. The squared factor 
loading in a row indicates the variance of a component accqunted for by 
the factor. Communality or represents this variance. The eigenvalue 
which is the sum of the squared factor loadings in a column indicates the 
amount of variation accounted for by the factor. As depicted in Table 10, 
all components of the Performance-Against-Standards Form are categorized 
as Oi,e factor and this factor explains 60 percent of the. total variance 
conta-ned in_the correlation matrix of Table 10. Thus, the Performance- 
Against-Standard Form is considered, to be a composite form containing nine 
highly related components and is considered as a composite criterion form 
and not a multi-criteria form. V ^'^'^ 

The pattern loadings in the promax rotated pattera matrix of Table 10 
can be explained as the measurement of the unique contribution or the 
dependence of each component on the factor. In other words, a pattern 
loading is viewed as a regression coefficient of the component on the 
factor. This coefficient may be treated as a weighted value assigned to 
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Table 10. Unrotated orthogonal factor matrix and promax. 

rotated pattern matrix on nine components of 
Performance-Against-Standard Form 



Component 
Categories 


Unrotated 
Factor 


0 

Orthogonal 
Matrix 


Promax Rotated 
Pattern Matrix 


Communality (h2) 




Factor 


Loadings 


Factor Loading 






I 


II 


I 




lA 


.85 


.11 


.85 


• 73' 


IB 


.73 


.25 


.73 


.53 


IC 


.'88 • 


-.06^ 


.88 


.77- 


ID 


-.73 - 


-.20 


.73 


.55 


IIA 


.78 . 


..16 


.78 


.61 


I IB 


-63 


.21 


.63 


■ .40 


lie 


.82 - 


..12 - 


•82 


.67 


IID 


.68 - 


-.28 


.68- 


.16 


HE 


.77 


.26 


-77 


. 60 



Eigenvalue:' 5.29 -35 

Total variance: 58.77 3.89 
n r'62 



each component. Data indicate that the factor loading of most components 
in Table 10 to be nearly equal. Providing weighted values to each compon- 
ent would have similar validity as providing an equal-level value to all 
components. 

Based on the evidence at hand, there is little reason to distinguish 
construct structures among the nine components. Even though one's job duty 
is correlated with each performance component as reported in AIR's study, 
this study confirms the Performance-Against-Standard Form to be a unidimen- 
sional scale which can^be represented by one overall construct. Thus, an 
average surfimated score of all items may be used to determine the rank of 
individual job performances. 
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Table 11. Listing of 60 Standards Included in the Performance- 
Against-Stlairdairdir^Tofm" Ry CQmponentr~and Item- Number •* 



Section 



Number of Standard 



Part 2. Results-Oriented Standards 
A* Program Planning Results * 

B. Program Promotion and 
Public Relation Results 

C. Program Implementation 
Results- 



D. Program Support Results 

Part 3. Behavioral Standards 

A. Program Planning Behavioral 

B. Program Promotions and 
Public Relations Behavior 

C Program Implementation 
Behaviors 

D. Program Support Behaviors 



Interpersonal arid Personal 
Behaviors Generally Related 
- to Jobt 



(IA) 12, 1, 7, ^ 

(IB) 35-/ 20, 32 



(IC) 51, 53, ^7, 66, 69, 

71, 125, ^2, 105, 100, 
111, 67, 89, 119, 118, 

^o^, 122, 83 

j(.ID) 132, 129, U^, 1^5*, 
139, 



(IIA) 2i|, 19, 6, 25, .1, 17 

(IIB) ^ 27, 3^, 38" 



(IIC) 68, 95, 81, 87, 70, 
98, 100, 115, "54 ' 

(IID) 117, 116, 123, 131, 
.126 

(HE) 176, 168, 19^, 192, 

200, 196, 172 



*See Appendix A for description of each standard. 
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FURTHER REDUCTION OF ITEMS • 



The Performance-Against-Standards Form was found to be a composite 
^measure-^ o-f employee performance rather than a multi-dimensional measure. 
Because the nine components of the performance appraisal form are signifi- 
cantly related to each other, agents' scores should not be compared for 
individual components. Since the form provides only one measure of total 
performance/ the number of items used to measure performance should be 
reduced. 

Using the item-total score correlation procedure for item selection 
(and described previously), each of the 142 items was correlated against 
the average total score. Based on the criterion of retaining items having 
Pearson product moment correlation coefficients of .50 or greater, it was 
determined that the item structure of the Performance-Against-Standards 
Form could be adequately represented by 60 items. In addition, Cronbach's 
alpha was calculated to determine the internal consistency of the 60-item 
scale. The resulting reliability coefficient of .96 indicates the reduced 
scale to be quite reliable. A listing of the item numbers by components 
for the 60 items is presented in Table 11. The distribution of items 
within each component is similar to that of the 142 item scale. The 
statements corresponding to the item numbers may be found in Appendix A. 

SUMMARY AND CONCLUSIONS ' 



This study has attempted to measure^ the degree of reliability and 
validity of the Performance-Against-Standards Form, as used in the personnel 
management system of the Ohio Cooperative Extension Service. Job analysis 
and the technique of item selection (in terms of item-total score corre- 
lation and/or item-criterion correlation) were used to judge item 
availability on the form and several items were eliminated from the form. 
Cronbach's alpha was utilized 'to measure the reliability of the form. In 
addition, multivariate measurement was used to determine the accuracy and 
relevance of the form. - 

The evidence indicates that the revised Performance-Agaijist-Standards 
Form does provide ai^ accurate and relevant measure' of employee performance. 
Thus, its inclusion in the performance appr'aisal system of the Ohio ^ 
Cooperative Extension Service is recommended. Because the Performance- 
Against-Standards Form represents a composite scale rather than a multi- 
dimensional scale, the number of items in the scale can be reduced from 
142. to 60. ' 

In conclusion, th.e Performance-Against-Standards Form has relevance 
for the Ohio Cooperative Extension Service in measuring the job performance 
of county extension agents. Additional research in different settings is 
required to confirm the overall applicability of the performance appraisal 
form for each State Cooperative Extension. Service. In addition, a long- 
range study should be designed that follows individuals from the time of 
application through several years of employment. 
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A. PROGRAM PLANNING RESULTS (Job Araiysis DutiM 1, 2, and 3) 

This wction Inciudm st$ndsrc/s for wmnwizktg th0 »g$nt's rmvfts in mming community 
n99ds, inyolv*ng ttm community In th9 fflMnning proem, prwptring ionfrnng^ trnd inmml 
pianSf pi9nning sp§clffe program 9ctivitim mi §vmits, 9hd rrmrmgirfg hts/h^ own tim$. 



Tabit M 
Rtftrtnet 



Stiodiird 



f IA12 



Hat good community raprtientition at planning stwons to sat county 
prioritiff. 



3.5 



IA1 



Idantifiad naw projact araas that intarastad many ciiants. 



2.5 



IA2 



Soma, but not all, of tha ntads of diantaia ara rafiactad in hit/bar 
situation statamants. 



1.5 



1A19I 



Occasionally doat a baraiy adaquata job bacauia of lait minuta ruth. 



0.5 



lA 7 I Pooriy organizad avants disiatitfy diants. 



-1.5 



iA18 



Tha numbar of minoritlai involvad at all lavalt of hit/bar program whara 
mmbi'itiai ara a part of tha aiidianca far axcaadt^ minimum raquiramants. 



3.5 



I IA16 



Has no difficulty racrutting advisory committaa mambari. 



2.5 



IA4 



Has an activa committaa fofj^astaasing^tha county situation, but tha 
committaa naads to ba mora raprasantativa of all sagmants of sodaty 
inthaaraa. " ^ - 



1.5 



IA17 



His/Har planning commlttaa funetlons but activttias naad Improvamant 
(a.g., provision for raplacamant or rotation of nrwmbars, or nwra ragular 
usa of committaa, or committaa naads to undarstand objactivas, raspon* 
sibiiitias and procaduras battar). 



1.5 



1A11 



Alianation axists batwaan planning committaa and agant. 



-1.5 



B. PROGRAM PROMOTION AND PUBLIC RELATIONS RESULTS (Job Anitytit Outy..1^^^ 
Thii motion indudm standard$ for wmmmiifng tfm ^gmrts' nmuits in promoting pni^ifm snd ihif 
Extmwhn Smne9, raiting fund$snd uting ti>9 rrmm mmik. 



18 36 


His/Har publidty matarials hava baan usad by othar countias in tha ragion. 3.5 












JB36, 


Raachas an audianca not larvad by Extarision'through a nawslattar. 3.5 * 












18 22 


Has obtaihad inaaaaad funds for spadal projacts by making wall 

praparad raquasts to tha county. 3.5 












IB 27 


Convincaa dubious county offldals about tt)a naad for a program. 2.5 












16 28 


Ha.{Sha is racognizad but.not wall kifibwn in county. i .5 












18 26 


Nawspapar staff hsMi to cut and rtwHta his/har matarial axtansivaly^ 0.5 












18 29 


Ailanatas diantaia and public by bacoming parsonally tnvolvad in conuo- 

varslal tssuaa in dty or county govammant. -1.5 












18 20 


Public officials or othar dadsion makan in thajraa axprass positiya 

opinions about tha agant and Extansion programs. 3.5 












IB 34 i Arrangas larga caramonial avants which run sn>o6thiy, ara wall attandad 

i and giva favorabia imprasstons to kay paopla. 3.5* 












IB 32 i Has good rapport with mass madia parsohnal so that Extansion activitiss 

j ara publlcizad. "3.5 
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C. PROGRAM IMPLEMENTATION RESULTS (Job Analysis Outies 4, 5, 6 and 7) 

This sKtionincludm standards for summarizing tht agant's rmutts'in conducting program' 
aetiynies. Than activitias includa broadaning tha communhy's involvamant in program , 
actlVltias: orxunizina and warkina vuith amnnf rmi'mWlmf ri^kjmlnnlr^ u., 



prrp^nn^ mjucMxiormi m9X9n9'%' givmg tonrmliZ9d instruction to groups; responding to disnt's 
rsqums for sdvics snd teehnicMi sssistsfKS snd using s^is/ist rs» ^ ' ^ 




f 


f 


r ^ 






Table A-l * * 
* Terence 

'^""^e'' . Sttndird- 




I 


; 2 
•i 1 

! = 


x 

)l 

ri 


c 


< 




* CI 

> 


His/Her prognm a«ivitit$ have rtsuittd in noteworthy change, in 
ciitnteie practicts. 


**,0 


1 
1 
1 
i 








- . — 


j IC53 


1 His/Her- results dtmonstrations havt allayed specific concernf of the public. 




H 1 

• ! 










4/ 

1 r- 


j Has developed a series of lesson plans aixi activities which has wide 
1 appeal to both youths a.id adults. 


3.5 






^« 


r 




. — 1 


j 1C66 


^ Community became aware of a public problem due to his/her program efforts. 






L 








• 


I 1C69 

1 


Additional sessions sometimes are set lip to LccomnxxJate overflow 
response |o agent's prograni activities. 


3.5 




1 — 

! 
1 










1 IC71 

1 


His/Her program projects have been successful* enough to be repeated 
in different parts of the country, ♦ 


3.5 






• 








|lC125 


^ ^ ; ^ 

\ His/Her ed.uoirional materials have been used repeatedly. 


3.5 




i 




' 1 • 




; IC42 


His/Her resuitt demonstrations have received good coverage from the media. 


2.$ 




\ 
\ 




I 




1C38 


Many of the attendees' questions are answered in his/her program 
prtsen canons. 


2.5 








J , 


' 1 


IC 56 


^ Has active programs, but does not branch out to reach new audiences. 


1.5 












1 


IC102 

1 


Members have expressed need for mere program activities but none 
have been offered by agent. 






\ 

1 










IC 57 i 


*Some of his/her program events have becoR)e annual events. 


3.5 




! 

,.1, , 








i IC76' 
! [ 


As a result of his/her educational activities, clientele gain a better 
unuvmanaing ot tne oasics. 


2.5' 




— i — 








1 IC99I 


Key community people give lukewarm support to programs. 


0.5 




v- 




7 




IC101 


Has persuaded reluctant community officials to serve on comminees. 


2.5 




\ 






. 1 


icilo 


Cofr.minee memben are confused and do not know vyfjat is expected of them. - 


-.05 




\ 








IC77 


All comminee niembers ire involved in carrying out at least one acti\ ty 
that was developed 'in the plan of work. 


2,5 




t 

t. 








IC95 


All his^er liaders are organizing or leading established groups. 


3.5 


f 
5 

\ 


1 

1 i 






IC105 


The impact of his^e^ leaders in program accomplishments is about 
^e same as that of the average agent. 


1.5 






i 
1 






I IC55! 


Has lost the interest of the volunteer leaders. 


-1.5 






i 






iciooj 


Has motivated and developed leadership in volunteers who were initially 
passive or difficult to work with. 


3.5 


j 


1 






ICIII; 


Has increased the amount of tiroeJeiiders volunteer. 

t (contintMd) 


3.5 


ill- 
1 ' 
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26 



♦ 

Tibit A-1 
Rfftftnce 

NumtMf Standard « 


t 

1 


X 

1 


i 


g 


z ^ 

o 


I 




j - 

1 Hm iittlo Of no prtmsturt turnovtr KDoriQ.voluntttr Itidtrs. 


o.D 




1 

i 










riss owi SOM to orttK sex or rmnortiy oimtrs in mt TiiiinQ ot voiunxttr 
ttadtTship positions. 


3.5 












1 1 


tC 67 


. -. Has high projtct complttions or accomplishmsnts by volunttor groups. 


3.5 












.; , 




1 ^>^Hit rtcruittd and dtvtlopsd Itsdtrs of ntw groups to a point whsrt ttit y ' 
'can function IndtptrYdtntly. 


3.5 










<» 




IC93 


F . ^ . -^-r^ 

Litdtfs havt lost rsspsct for tht agtnt. 


-1.5 














1C96 


VoluntHf groups can rtiy on hts/har. to h«lp tht m prtpsrt for tvants. " 


2.5 














IC96 


Good voiuntatr iaacitrs haya-rasigr>ad'in protast ovar agant'$%ctions. 


-1.5 














IC116 


Voluntaare gain laadarshipaxpariancas. 


3.5 






- 






> 

i — 1 


• IC72 


Adoption of 1 naw practica racommandad by agant has axpandad from a 
faw local innovatoa to batoma commonp<aoa among tha ciiantala group. 


.3.5 














SC119 


Qiants hava^ma to'dapand on tha agant's naWsltttar for in*dapth information. 3.5 












' ' 


MC73 


Saw naad for and davalopad a tachrHcal ntwslattar for which x^rnn 
davaiopad a long mailing list. 


3.5 














IC 54 


. Has baan rasponslbia for savarai parsons vanning county or nata awards. 


2.5 














IC118 


His/Har aducationarprojacts hava mada a significant impact on tha 
local community. 


3.5 












1 1 


1C104 


Gats initially hasitant parsons to participata succasifully in a projact. ^ 


3.5 












' ' 


' IC122 


Has ovarooma initial lack of trust of a diarttaia group in a program a^ivity. 


3.5 


■ 












' 1C83 


His/Har programs ara'vary popular with local paopla. 


3.5 














JC128 


Is rsoognizad by ditntala groups as having tha skills to assist tham. 


3.5 














ICK 


Is sought aftar as a spaakar for araa and stata programs on cartain subjacts. 


3.5 












1 — 1 


IC106 


Gats othar agandas to cut rad.tapa in ordar to accomplish program objactiva. 


3.5 










<? 




IC90 


. Has gottan othar ralayant public agants in araa to cooparativdy. plan 
. an activity for tha community. 


3.5 














IC9T 


Got savarai raiavant agandas to shara raaourcas and staff rasponsibilitias 
in providing a succassful naw prognini for diantala. 


3.5> 














IC78 


Utilizas tha halp of a faw spadallsts, but thair maximum contributions > 
ara not raaiizad. 


0.5 








•i 






; I C124 

i 1 


Has chosan rasourca parsons to spaak whosa lack of practical axpartanca 
disaaditad tham in tha audianca's ayas. 


-0.6 








— r 

t 







27 
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PROGRAM SUPPORT RESULTS {Job Analysis Duties 8, 9, 12 and 13) 
^ This section Includes ^ndsrds for summarizing tha sgmrt's rmuits in rmrfing $nd •vwiuatinQ 
t9sics and continuing prof agonal dm^opmmt ^'ntm^fmuww 



T«bl« A.t 

Reference 
Number . 



Standard 



i * 



< 

Z 



ID 132 



Kwprgood records and mtmos to capitally upon tht ntxt tvtnt or for 
ntxt staffs mtmbar to ust. * 



3.5 



i ID 129 



! ID145I 



His/Htr rtports indtcatt pro9ranf» prograts with tingibla facts such is 
numbars, dollars, or dafinita changts in initudas, knowltdga, or skill. 



3.5 



ID 1441 His/Har rtports ara on tima and wall dona. 



2.5 



Immadiataly following activity, ha/sha complatfi tha nacassary avaluation 

forms. 



2.5 



ID 136 



Has about as many graduata or pxmrrm school cradits as tha avaragt 
agant with tha sama tanura. 



1.5 



i ID 135 



Involvasfawvoluntaars and local laadars in avaluating programs* 



ID 141 



ID 140 



0.5 



Shows ilttia or no ungibla tvidanca of profassional growth aftar training. 



-0.5 



Filas ara in fair condition. 



2,5 



j ID 143 



Is well ragardad for making^parsonal saaificas for profassional 
self iniprovamants. 



' I '0 139| Cmtn gtti fnarly 100% rtturw from prticipanti to hit/W iiyilmtion formt. 17 



28 
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A. PROGRAM PLANNING BEHAVIORS (Job Analysis Duti« 1, 2, and 3) 



28 



rmds, invoMng th€ community in tht pining proem, prtptring long-nng^ $nd snnM 
pkms, pisnning specific prognm sctivftim and mmnts, and managing his/har own tima, 

Tabi« A.1 
Rtfcrtnce 

Nun*er Standard 




1 


s 
1 




2 

2 
c 

& 


< 








I 


Z 




1 )IA24 


Thoroughly plans prograrm with txttnstvt iMVolvt mtnt of organizad advisory 
groups, community Itadtrs, public officials and rtprasantativts of inttndtd 
audiancis. 


4.5 








> 






ilA19 


Thinks through aach month, aach wttk, aach activity and knows by ittm 
whit natds to bt dont and whtrt aytryont will ba or should ba. 
















IIA18 


Amtopttas n«w otanxma ana actively invoivas tnam in tnt planning procass. 
















1 I1A9 


Ofttn usts formal survtys to ditarmina community netds. 














1 — 1 


i 

IIA6 


Plan of work soimtimaa nMds to sptcffy mora dtarly Itadarship and 
staff involvtmtnt for impitmanting plan. 


1.5 












1 1 


ItA A 


Pta If--- |-ihiTtiiii^ htit mrm snriMtirfiM nAt mifall dflffnttri sc to 
uaiai iiiinvs oojwcuvaa, ouvuwj atw luiiwiimw nwi wwii uvmiivu w 

audianct, contant and axpictad bahavlor. 


0.5 












1 1 


!iA3 


His/Har plan of work is only partially baaad on probltms and obitctivas 
sttttd in long-f anga plana and situational statamants. 


0.5 














MA 10 


Writas a plan of work, than ignoras tha plan and doas unralatad prognms. 


.0.5 














ilA25 


Saas'total pictura -stata, county, and loca!*^in his/har program planning. 


4.5 












1 1 


IIA21 


Idantif iaa a spadfic targat audlanca in planning programs. 


3.5 














MAS 


Providas summary reports of pravious yaars activitias at planning maatingi. 


2.5 














HAS 


His/Har plan of work is aimad at a somawhat iimitad sagmant of tha 
county population. c 


0.5 














IIAI 


Doat net fully astablish prioritiaa. 


0.5 














ifAII 


Is haphazard in organization of avants, dofrg only what has to ba dona. ^0.5 














IIA17 


Plans a program that is timaiy ahdimttt ^ naads of tha paopla. 


3.5 















B. PROGRAM PROMOTION AND PUBLIC RELATIONS BEHAVIORS (Job Analysis Duty 10) 
Thk metion Includm mndatdt for wnnrn^ng thttgmk'tbthwionjn promoting prognim ind 



11827 


Maintains good ralatiom with iooal officials and tha powar structura. 3.5 












lis 34 


Has wrfttan human intaraat storlas to highlight aapacts of Extanaion wortc. 2.5 












-IIB37 


Maintains communieetion with soma local laadars, organizations and groups. 1.5 












lis 38 


Efforts toaaaiat with,planninga?>d implam^^ 

programs lack eonaistaney. 0.5 








— 




, 11643 

1 


Thara ara soma parts of tha program that ha/sha saas as baing unimportant 

and thus will, noi ba concanrwd with tham. -•0.5 












; IIB32 


Makas original uaa of mau madia. 












: HB36 


Is invoivad in at laatt ona'community activity, or organization not 
ralatad.tb.Extanaion. 2.5 


1 











54^ 



29 



29 



Table AO 

??e*erence 

StindHfd 


1 ^ I 

^ 1 1 1 -s 
^ • c * z 


1 




! 

\t\\B 39 { M«kt$ insufficitnt u$e of ont or more of the mass media. " 0.5 


.1 
i 








JIB 44 ! Shows.disregard for local values and customs. «i 5 

h- 1 : — , 


I ! 

\ i 








\ iiB 36 1 Has developed a county-wide mailing list of interested individuals. 2.5 


\ i 1 


1 
1 





C. PROGRAM IMPLEMENTATION BEHAVIORS (Job Analysis Duties 4, 5, 6 and 7) 



T^^^J^^Z^^^J-' '^"'•"9 r'^, b^mnon iff conducting prognm sctivhim. 
w^i^^hlJ^^^J^*-T^'J^ commt/n/fy'* mvohtnmt in program Ktivitim; orgtnizing and 
1Z fLl?,X J?r^L^,^''^' ''•^'°Pi"8'"duang tylmdms. preparing tduettionai r^tttru/i- gi^ 

> 59 i Continually lo oks for and finds' ntw auditiicts to work vwth. 3.5 j ~~ 



lie 59 


! Continually looks for and finds new audiences to work with. 

"1 '* ' 


3.5 














IIC68 

t 


j Exhibits innovative teaching techniques. 


3.5 














j lie 73- j Almost never uses "cannid" materials for an audience. 


2.5 














IIC71 


I Writes teaching goals and outlines lesson plans.to accomp*ish the goals. 


2.5 














IIC95 
; 


t — — ' — - — 
i Does a fair job of teaching in assigned subject matter area. 


1.5 














1 ricsi 


1 Effectively uses only a portion of twchlng methods. 


0.5 














lie 78 ! Takes for granted people understand what he/she is saying. .0:5 






1 








^ lie 72 


1 Presents information already known by the audience. 


-0.5 






i 








IIC43 

1 


! Shows prejudice in dealing with minorities and Jo w-income audiences. 


-1.5 






-4 








j lie 87 


j Does thorough research before conducting method demonstration. 


3.5 












1 1 


nC92 


Teaches most recent information on the subject. 


3.5 






i 






lie 70 


Always does some twching in every educational activity for which 
h#/8he has responsibility. 


2.5 




! i' 






Iie49 



Considers time, place and other drcumstances in organizing a group. 


2.5 




■I : 






iie64 


Conducts programs in the county in all locations to- meet the needs of all 
people, irrespective of race, color, national origin or economic circumstances. 


3.5 




! 
1 

•! 


f 






1 iie53j 

! ! 


Requires prodding to get him^ef to expend program to cover all major 
interests and enterprise. 


-0.5 




: i 
i ' 






r iie98 

!. .1 


Uses imaginative and appropriate techniques to persuade the client to 
. try or to adopt recommended practlow. 








i 
i 

t 






iieiooi 


Introduces innovations bf.^ a trial or demonstrative basis, selecting 
and finding cooperators carefully. 


3.5 






i ■ 






fiicnsi 


Provides accurate, complete, andcurrent information or assistance. 


3.5 


i 








iieiioi 


Delays.rtcontacting a client. 

Icontinued) ' 


0.5 . 


i i 


1 

5 I 
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Table A-1 
Reference 

Number Standard 


% 


1 

C 


1 

Z 


\ $ 

! ^ 

1 C 


< 




1 

jllCIOt 

i 

» 


When he/she receives call for technical assistance, makes excuses for not 
providing assistance requested. 


-0.5 














nicios 


Gives information from "top of the head" - often not documented 
by research. 


-1.5 














liC103 


Doesn't take seriously the questions, problems, or concerns brought by clients. 


-1,5 














i)C106 


Gives generalized answers to callers with specific inquiries. 


0.5 












' ' 


1 IC 11 2I Rtcommtnds ntw products only aftt r adtquats triiii. 


2.5 












I 1 


lie 54^ 


Devotes personal individual attention to new leaders, counseling and 
advising them and strengthening their abilities and confidence. 


3.5 














ilC46 


Obtains new leaders without conferring with other suff members who are 
acquainted with prospective leaders. 


0.5 












1 ' f 


lie 47 


. - 

Recognizes luden with only a few methods of recognition. 


0.5 












i 1 


lie 62 


Aaks volunteers to do things at the laet minute. 


-0.5 








I 
1 
1 




lie 55 


involves a few volunteer and local leaders in impienienting programs. 


0.5 












, J 


lie 58 


Provides le;)ders wtih opportunities to serve and exercise their leadership 
skills in lirrited areas. 


0.5 














lie 60 


Does a fair job of identifying and recruiting local leaders. 


1.5 














lie 76 


Uses' specialists and resource people to maximum advantage. 


3.5 












1 J 


lie 69 


Gives specialists ample notice so they can work meetings into their 
schedules. 


2.5 














lie 79 


Is unwilling to teach subject areas, relying totally on specialist 
other resources. ^' \\ 


-0.5 














lie 102 


Has a breadth and depth of knowledge about a ranpe of specif icpatt, 
methods and pn'ndpies. / \ 


3.5 















D. PROGRAM SUPPORT BEHAVIORS (Job Afiaiysts Dutte 8, S^pnd 13} 

This Mction mdudm stanckrxk for sumnwrlzing thi $g0ntfs b^tmvion in reporting and mmluating 
prognm Ktivitm and accompliihnmnn; pmiorming off tea managanmt Aff adminiatrati^ taaks 
and continuing prof aaaional da^ahpnmnt 



110 11 tI Develops evaluative materials for program during planning stage. . 3.5 












! _^ 

IID'116| Measures accomplishments in terms of behavioral objectives. 3.5^ 












'no 135! Gives examples in reports of changes in behavior. 3.5 












: \ ' 

110 12a Evaluates program offerings in a variety of ways and documents results. 2.5 












1 IID120j Selects certain program spals for in-depth evaluation uf ing surveys and 

i j instruments designed for that specific program. 2.5 













3i 



31 



Tabte A-l 5^ , _ . 

. Standard 




1 0 1 f 5 3 


< 

z 




iII014< 


Oj Includes a viritty'of tabits and graphic mtans of prtsenting 
J mtaningful infornrwtion. 


2.5 




! ! 1 

i i 




T ' 

1 ' -in. nil- ' 


illO 12 

i' 


I Hls/Htr program tvaluations art not toully inciusivt of activitin 
raporu, evtntt, and committtt work throughout tht yar.' 


0.5 






!' 

i 


5 ■ 
I 






illD 127 


' Evaluatfs prograrm basad mainly on his/htriobMrvat:ons. 


0.5 




1 » 

1 1 




( 1 


!IID 151 


1 Oot$ not kttp an accurate record of all office callers. 


0.5 




! '\ 




J . ■ 


!|ID13C 


' . His/Her evaluations consist mostly of a list of agent inputs, activities 
1 performed, and amount of partictpetion. 


0.5 


! : I 






jllO 144.i Oo«s not takt advantagi of profastidnal socittias of which ha/sha 

j ! * tntmbtr for profassional improwamant. ^ 


0.5 


• 1 




1 1 


1110128 

i 


1 Ooes not set behavjpra! objectives as a means for evaluating programs. 


-0.5 


J 








liD ^AQ 


Mas tar fewer credits in graduate or summer school than most agent: 
with same amount of tenure. 


-0.5 


. 1 • 




i 






110155 


j Is conKientious about the office appearance at all times. 

r — — ^ — ^- 


2.5 


1 ! 








no 132 


Interprets statistical data appropriately. 


2.5 










110160 


» Routine office jobs are not performed efficiently. 


0.5 




1 








110125 


Evaluates resulu at time of annual planning rather than making 
evaluation a continuous process. 


0.5 




I - 

t 
1 






1 ( 


1110 145 


He/She re^aluates his/her job and program performance at periodic . 
intervals during the program year. 


3.5 












1 J 


110119! 


' When no fornuil evaluation is posn'ble, follows up with clients and/or 
peers to^ discover strengths and weaknesses in program. 


3.5 








— 






110157 


. Lau suppliee of bulletins and other needed information and ^ 
materials run out. 


0.5 ■ 












' '. . ■ 


110133 


^ Writes long reports .without' really saying anything important. 


-0.5 














110147 


Reeds, studiee, and learns from a variety of sourcaa. 


2.5 






! 








110141 


Participates in professional improvement activitiee whenever possible. 


2.5 














110131 


Keeps accurate records of accompiishmants for reporting purpoees. 


3.5 . 






1 








110.126 


Has an effective My of questioning and observing clients to determine 
how much technical information is being used or if attitudes are changing. . 


3.5 






1 

1 
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E. INTERPERSONAL AND PERSONAL BEHAVIORS GENERALLY 
RELATED TO JOB (Job Analysis Duty 11. and Job Ptrvasivt) . 



This $$ction include standards for wmrmuizing tfm sgmtt^s inmprnwrml MOd f>m9on$l 
b9h9¥ion Mnd chmctfitdcs that art garmaily rafatad to jot mpomibilitiaL 

Tatow A.I 
Rcftrtnce 

'Numbtf StifxJafd 


1 


8 

i 
S 


1 
1 


1 

c 

& 


< 
Z 


j 


'tiE176<^ Dull with difftojlt inttrptnonil probltm by bringing tht indivlduali 
togtthfr to tilk it outv 


3.5 














J1IEI68 


Kmpi co-^rktrs inforrhtd of actioni that affact tha antifa offiot. 


3.5 














jtlE194 


' ' Maintains a proifauiorMl Imaga at all timai. 


. 3.5 














MIE161 


Takn tima to hafp a o^workar with a probiam. 


.,3.5 














i ^ 

j:IiE'199 


Caras lass about ialf*raoognitioa and praiia than about maating.tha 
naadrdf tha pippla. ": 


3.5 












^ ^ 


I1IE219 


Has innovativa idaas, but doat not try to foroa changa. 


2.5 














t ■ • . ' 

UlE220i ' Arnm:at work on tim. 

1 


2.5 














nE189 


Accapts faiiura without placing blama on othan. 


2.5 












, , 


liE221 


Graats a nawjdaa by looking for its inadaquadaa. 


0.5 














IIE187 


Trias to take sola cradit for whatavar is accompltshad. 














, , 


T 

1IE1M 


Rafars eompliniants haard about othar staff. workars outsida tha offica 
back to tha parson. 


2.5 












1 1 


HE 191 


Ooasn't try to "snow" paopla. 


.„2.5 














ilE222 


Is lata for maatings or activftias. 


r1.5 














HE 192 


Enjoys working with paopla. 


2.5 














HE200 


FoHowi up on contacts and/or commrtmants. 


3.5 














HE 178 


Allows othars to racaiva racognition avtn whan ha/sha is tha main 
raason for tha achiavamant. 


3.5 














11E196 


■ ' 1 — — ' ' 

Radiatas anthusiaam for job. 


3.5. 














HE 172 


Halps motivata paopla of all lavals. 


3.5 














HE 174 


Handias conflict calmly. 


3.5 














HE 208 


Works irragular and axtra hours whin nacassa'ry. . 


' 3.5 















33 ' ' 

A. SUPERVISORY PERFORMANCE (Job Analysis Duty U) 



This stction include standtrds for summarizing the supervisory ptrformsnct of agrnits 
who hM official responsibiltty for suparvising othar employaas. 

Tabie A-1 
flererence 

Numoer Standard 


«> 
« 

< 




Nsilhar 


X 

5 
6 


< 
z 




: III 14 


Isn't too busy to converse with subordinates no matter how small the 
deulls. 


3.5 




j 


1 

i 




• 


^ III40 


j . Explains to subordinates why resources aren't available Instud of 
1 simply saying "NO". 


3.5 




T i 

I 

I 

\ - 






( 1 


{ III 41 


J Allows time for discussion of plans and accomplishments. 


2.5 










I 1 


, III 9 


Listens to subordinates' complaints. 


* 2.5 










, , 


' III 10 


Hei|M coordinate work with secretary* 


2.5 




\ > 








i H1 11 


! Has adequate resources for a besic program. 


1.5 




i 






1 J 


; III 13 


Reports programs and activities to subordinates when asked. 


1.5 






1 

I 






I III34 


Overrules ideas that subordinates have without. giving. the idea a fair trial. 


—0.5 










1 .J 


'1139 


Fails to leave or provide complete and menmngful instructions to 
subordinates. 


-0.5 




1 
1 








! Ill 7 


-> 

Plavs down 'the ideas of subordinate! 


— 1 ,0 




1 
i 








111 18 


Doesn't discuss problems with office staff who could have some insight 
for solving the problem. 


-1.5 




1 








III 26 

1 
1 


Makes time to ^ive direct supervision to Ins experieficed or leu 
capable agorro. 






i 

I 

i 






I J 


III 25 1 


Holds office conferences to coordinate activities of ad staff. 


3..5 






i 






[ „,! 


; III 6 1 

t * 


Updates subordinates on all program and events as the need requires. 


2.5 












1 1 


111 38 


Waits till last minute and then expects the secretarin to do the 
needed work. 


-0.5 






1 


i 
1 
1 




• ■ 


11128 


He/she tries to do everything rather than delegate some of it. 


-0.5 














11119 


TakM sidee ia disagreenrwnts among subordinates. 


-1.5 














III 15 


Helps subordinates get equlpmem and riepurces needed. 


3.5 














11124 


Seaeurial work load is planned suffidentiy in advance to allow for 
both priorities and routine needs of staff. 


3.5 














III 30 


Assesses needs of individual staff workers in office. 


3.5 












L J 



>' 
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SUMMARY OF RESEARCH SERIES 



Organizations use different methods in evaluating their personnel. 
Most agencies have been seeking better ways to appraise their 
personnel so that proper Secisions'^ can be made about merit pay, pro- 
motions, and related matters. The Cooperative Extension Service 
contracted the development of procedures for' evaluating personnel 
performance based upon standards. This study testjs the rc;liability * 
and validity of those procedures. It should be of interest to 
Extension administrators and others interested in performance 
appraisal of personnel. / ' ' 

This summary is based on a Doctor of Philosophy dissertation by Kun 
Sun Shiao under the direction of Howard Ladewi^. Dr. Shiao- is an 
Associate Professor, Department of Agridulture Extension, National 
Taiwan University, Taipei Taiwan. Dr. Ladewig is an Associate 
Processor, Department of Rural Sociology, The Texas A&M IMiversity 
System, College Station. Special appreciation is due Richard E. 
Young, Professor, lister of Adult and Continuing Education Program, 
Washington State University; ^Clarence J. Cunningham, Assistant 
Director, Administration, The Ohio State University; and Fred J. 
Peabody, Professor, Cooperative Extension Service, Michigan State 
diversity for their c^pitical review of this manuscript prior to its 
publication. _ ' " . 

Research has been an important function of the'' Department of Agricul- 
tural Education since it was established in 1917. Research conducted 
by the Department has generally been in the form of graduate theses, 
staff studies and funded research. The purpose of this serines is to 
make useful knowledge from. such research available to practitioners 
in the profession. Individuals desiring additional information on 
this topic should examine the references cited. 
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